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EXECUTIVE  SUMMARY 

CAE  Inc.  was  contracted  (Solicitation  No.W7707-135643/B)  by  Defence  Research  and 
Development  Canada  (DRDC)  to  conduct  a  literature  survey  regarding  the  use  of  OpenSIM  or 
similar  Virtual  World  (vWorld)  technologies  to  develop  and  review  system  designs  such  as 
compartment  layouts  and  team  processes  on  naval  platforms,  particularly  when  stakeholders 
from  the  Department  of  National  Defence/Canadian  Armed  Forces  (DND/CAF)  are  distributed. 
For  the  purpose  of  the  literature  survey  reported  herein,  a  vWorld  is  a  computer  graphical 
representation  of  three  dimensional,  physical  environments  that  users  may  interact  with  by 
controlling  computer  generated  agents  (often  referred  to  as  avatars)  or  with  agents  controlled  by 
other  users1.  The  scope  of  the  literature  survey  was  to  focus  on  the  use  of  OpenSIM  and  to 
discuss: 

a.  Convergent  and  divergent  trends; 

b.  Active  user  community  and  applications; 

c.  The  use  of  OpenSIM  in  support  of  system  development  for:  Navy  operations,  Navy  training, 
other  military  operations  and  training,  real-time  defence  and  security  (e.g.,  emergency 
operations)  or  other  real-time,  operational,  complex,  high-risk  environments; 

d.  How  OpenSIM  can  be  linked  to  other  analysis  tools;  and, 

e.  How  OpenSIM  has  been  linked  to  other  applications  (CAE,  2013b). 

Only  10  documents  were  reviewed  in  depth  due  to  constraints  on  the  level  of  effort  of  the 
contract;  75  abstracts  were  reviewed  to  determine  a  broader  impression  of  the  vWorld  domain. 
Only  a  few  of  the  documents  referred  to  OpenSIM  explicitly  and  none  provided  a  great  deal  of 
detail  on  how  each  OpenSIM  application  was  created;  instead  the  reviewed  reports  focused  on 
the  use  of  vWorlds. 

The  conclusions  of  this  work  are: 

a.  Several  vWorld  platforms  are  available  that  could  be  and  are  being  applied  to  military 
domains  (at  least  on  an  experimental  basis)  including:  Second  Life,  OpenSIM,  Virtual  Battle 
Space  2  (VBS2)  /  VBS3,  Active  Worlds,  Croquet,  On-Line  Interactive  Virtual  Environment 
(OLIVE),  Open  Colbalt  Alpha,  OpenEnergySim,  Open  Wonderland,  Pivote  and  Teleplace; 

b.  Converging  trends  reveal  that  OpenSIM  is  being  demonstrated  as  a  capable  and  flexible 
platform.  There  is  an  active  OpenSIM  user  community  in  the  application  domains  of  interest 
to  DRDC  (Section  3.2)  including  organizations  such  as:  US  Naval  Undersea  Warfare  Center 
(NUWC),  Australian  Defence  Science  and  Technology  Organization  (DSTO),  the  US  Navy 
Postgraduate  School,  Florida  Institute  of  Technology  (Harris  Institute  for  Assured 
Information),  University  of  Edinburgh  (Artificial  Intelligence  Applications  Institute),  and  the 
US  Army; 


1  Summarized  from  Wikipedia:  http://en.wikipedia.org/wikiA/irtual_world 
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c.  OpenSIM  can  support  diverse  applications  including  systems  development  and  training,  and 
the  number  of  institutions  exploring  vWorlds  in  general  seems  to  be  increasing  as 
awareness  of  the  technology  increases.  However,  the  scientific  literature  assessing  the 
validity  of  vWorlds  for  training  or  analysis  is  immature  and  the  effectiveness  of  these 
technologies  remains  to  be  determined; 

d.  Research  and  development  or  at  the  very  least  knowledgeable  analysis  is  needed  to  match 
training  requirements  to  evolving  vWorld  capabilities  in  order  to  establish  the  validity  of  a 
vWorld  for  the  intended  training  application.  Much  of  what  is  known  about  the  effectiveness 
of  simulation-based  training  should  be  applicable,  but  there  are  many  misconceptions  held 
by  education  practitioners; 

e.  Basic  research  is  needed  to  develop  open  architecture  (OA)  frameworks  for  specifying 
vWorlds  to  promote  exchange  and  reuse  of  models  as  well  as  reduce  the  incidence  of 
models  that  cannot  be  exchanged  between  virtual  environments  (Scacchi  &  Alspaugh,  2008; 
Scacchi,  2012);  and, 

f.  OpenSIM  can  be  linked  to  other  analysis  tools  through  an  Application  Programming 
Interface  (API). 

Recommendations  are: 

a.  Employ  a  user  centred  design  approach  to  develop  tools  fit  for  use  in  vWorlds  for  the 
purpose  of  systems  development  and  training.  In  accordance  with  guidance  from  MIL-STD- 
46855A,  the  Mission,  Function,  and  Task  Analysis  (MFTA)  methodology  has  been  widely 
used  within  the  Human  Factors  community  to  conduct  analyses  of  user  requirements  for 
over  twenty  years.  Suggested  capabilities  derived  from  the  reviewed  literature  that  should 
be  provided  by  a  vWorld  to  support  systems  development  and  training  effectively  are 
documented  in  Section  4.1.2  and  Section  4.2.2,  respectively; 

b.  Conduct  a  number  of  use-case  experiments  that  compare  vWorld  development  to 
conventional  design  methods  to  assess  the  relative  costs  and  effectiveness  of  immersive 
environments  for  design.  The  use-cases  should  be  sufficiently  challenging  that  they  can 
discriminate  between  the  design  approaches,  but  limited  in  scope  so  that  they  could  be 
conducted  in  an  affordable  manner.  Further,  the  experimental  design  should  explicitly 
include  adequate  training  in  both  traditional  and  vWorld  design  methods  to  minimize  biases 
due  to  learning  the  methods  per  se  (Scipione,  2006); 

c.  Employ  a  disciplined,  instructional  design  approach  before  considering  vWorlds  as  a 
potential  solution  for  training  delivery; 

d.  Conduct  a  directed  investigation  of  training  effectiveness.  Effective  training  typically  requires 
a  disciplined  application  of  instructional  design,  applying  what  is  known  about  how  people 
learn,  how  learning  can  be  disrupted  and  how  technologies  support  learning;  and, 

e.  Validate  the  use  of  OpenSIM  for  learning  and  training  by  employing  transfer  of  training 
analysis  methods  as  prescribed  by  AGARD  (1980). 
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1  INTRODUCTION 

This  literature  survey  report  was  developed  by  CAE  Inc.  (CAE)  for  the  Defence  Research  and 
Development  Canada  (DRDC)  -  Atlantic  project  entitled  “vWorlds  Capability  Development 
Support”  under  contract  #W7707-1 35643/001/HAL. 

1.1  Background 

Virtual  Worlds  (vWorlds)  are  interactive  simulated  environments  accessible  by  multiple 
distributed  users  through  an  online  interface.  Human  representation  typically  takes  the  form  of 
human  avatars  that  are  computer  representations  of  agents  that  the  user  can  control  to  interact 
with  the  virtual  environment2.  vWorlds  have  seen  increasing  usage  for  several  purposes.  The 
largest  and  most  common  use  of  vWorlds  is  for  commercial  gaming  or  Massively  Multiplayer 
Online  Role  Playing  Games  (MMORPG)  such  as  World  of  Warcraft  and  Everquest.  However, 
vWorlds  have  also  seen  application  in  the  domains  of  education  and  training,  community 
building,  tourism,  and  e-commerce.  For  commercial  applications  that  seek  a  world-wide  market, 
persistence  of  the  vWorld  is  an  important  feature.  Persistence  entails  that  vWorlds  endure  (as- 
is)  when  powered  off  and  on  again  without  the  need  for  a  user  to  explicitly  save  the  vWorld 
state.  For  specific  military  applications,  persistence  may  not  be  an  important  feature  as  much  as 
distributed-user  access  and  so  this  review  was  not  constrained  to  virtual  environments  that  are 
generally  considered  to  be  persistent. 

Defence  Research  and  Development  Canada  (DRDC)  is  interested  in  the  use  of  vWorlds  to 
enable  the  Department  of  National  Defence/Canadian  Armed  Forces  (DND/CAF)  to  develop 
and  review  system  designs  such  as  compartment  layouts  and  team  processes  on  naval 
platforms  (Solicitation  No.W7707-135643/B). 

1.2  Objective 

The  objective  of  the  contract  is  for  CAE  to  provide  support  to  a  two  year  agility  project  to 
develop  a  capability  to  use  vWorlds  at  DRDC  Atlantic  (Halifax,  NS)  and  DRDC  Toronto  and  to 
demonstrate  the  application  of  vWorld  technologies  to  naval  projects.  The  support  is  required  to 
cover: 

•  Provision  of  training  on  the  Open  Simulator  (OpenSIM)  vWorld  technology  (CAE,  2013a); 

•  Conduct  of  studies  of  the  interoperability  of  OpenSIM  with  other  DND/CAF  simulation 
technology  (CAE,  2013b);  and, 

•  Conduct  of  a  literature  survey  (reported  herein)  intended  to  inform  DRDC  as  to  how 
OpenSim  or  similar  vWorld  technology  could  be  used  to  support  activities  such  as  concept 
development,  collaborative  design  review,  and  experimentation  for  operations  rooms  in 
naval  platforms. 


2  Summarized  from  Wikipedia:  http://en.wikipedia.org/wiki/Virtual  world 
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1.3  Scope  of  Literature  Survey 

Following  submission  of  a  draft  report  (Version  01),  the  objectives  and  scope  of  the  literature 

survey  was  discussed  between  DRDC  and  CAE.  The  scope  of  the  literature  survey  was  revised 

and  requested  by  DRDC  to  be  as  follows: 

•  Convergent  and  divergent  trends; 

•  Active  user  community  and  applications; 

•  The  use  of  OpenSIM  in  support  of  system  development  for:  Navy  operations,  Navy  training, 
other  military  operations  and  training,  real-time  defence  and  security  (e.g.,  emergency 
operations)  or  other  real-time,  operational,  complex,  high-risk  environments; 

•  How  OpenSIM  can  be  linked  to  other  analysis  tools;  and, 

•  How  OpenSIM  has  been  linked  to  other  applications  (CAE,  2013b). 

1.4  This  Document 

The  structure  of  this  document  is  described  as  follows: 

•  Section  1:  Introduction.  Presents  an  overview  of  the  contract,  objectives,  and  scope  of  this 
report; 

•  Section  2:  Method.  Presents  the  method  used  to  conduct  the  literature  survey  including  the 
approved  keywords  by  DRDC  and  documents  reviewed  by  CAE  for  both  the  original  and 
revised  scope  of  work; 

•  Section  3:  Results.  Presents  the  results  of  the  literature  survey  including  examples  of 
vWorlds  platforms,  an  overview  of  the  documents  reviewed,  and  known  users  and 
applications  of  the  OpenSIM  platform  in  the  areas  of  interest  to  DRDC; 

•  Section  4:  Discussion.  Presents  a  discussion  of  the  common  issues  and  requirements  of 
using  the  OpenSIM  platform  for  systems  development  and  training.  Practical  solutions  are 
then  proposed  to  address  these  issues  and  requirements; 

•  Section  5:  Conclusions  &  Recommendations.  Presents  concluding  remarks  and 
recommendations  for  follow  on  work; 

•  Section  6:  References.  Presents  a  list  of  cited  references; 

•  Section  7:  List  of  Acronyms.  Identifies  the  acronyms  and  abbreviations  used  throughout 
this  document;  and, 

•  Appendix  A.1:  Search  Results  (Original  Scope).  Presents  the  search  results  for  the 
original  scope  of  work. 
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2  METHOD 

This  section  documents  the  method  that  was  used  to  conduct  the  literature  survey.  The 

following  approach  was  taken: 

•  Prior  to  conducting  the  search,  the  CAE  Team  collaborated  with  the  Scientific  Authority  (SA) 
to  agree  upon  the  process,  the  dimensions  (i.e.,  keywords  and  databases)  and 
responsibilities.  The  SA  provided  direction  regarding  key  literature  to  be  reviewed; 

•  Conducted  the  search  of  publicly  accessible  databases  using  approved  keywords; 

•  The  search  of  publicly  accessible  databases  using  the  keyword  combinations  led  to  large 
numbers  of  potentially  relevant  literature.  Accordingly,  criteria  was  developed  and  used  to 
reduce  the  number  of  relevant  findings  in  the  literature  survey;  and, 

•  Approved  references  were  subsequently  reviewed. 

2.1  Keywords 

The  following  keywords  were  validated  with  the  SA  and  used  to  conduct  the  literature  search: 


Applications 

•  Modelling 

Army 

•  Multi-National 

Challenges 

•  Navy 

Collaboration 

•  Open  Simulator 

Command  and  Control  (C2) 

•  OpenSim 

Command  center 

•  Operations  (Ops)  Center 

Command  centre 

•  Operations  (Ops)  Centre 

Concept  development 

•  Operations 

Control  room 

•  Ops  room 

Center 

•  Requirements  Analysis 

Centre 

•  Research 

Defence 

•  Second  Life 

Defense 

•  Serious  Games 

Design  review 

•  Simulation 

Education 

•  Success 

Experimentation 

•  System  analysis 

Human  Centered  Design 

•  System  design 

Human  Factors 

•  System  evaluation 

Human  Systems  Integration  (HSI) 

•  Team  decision  making 

Joint  Interagency  Multi-National  and 

•  Teamwork 

Public  (JIMP) 

•  Training 

Joint 

•  Unity 

Lessons  Learned 

•  Use  cases 

Military 

•  Virtual  Worlds 

Modeling 
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2.2  Documents  Approved  for  Review 

This  section  presents  the  documents  that  were  approved  by  the  SA  for  review  by  CAE.  The 
complete  search  results  are  presented  in  Appendix  A.1. 

A  total  of  75  items  were  identified  for  potential  review  and  each  abstract  was  evaluated  by  the 
SA,  identifying  the  following  29  documents  as  high  priority  based  on  their  relevance  to 
supporting  concept  development,  collaborative  design  review  and  experimentation  for 
operations  rooms  in  naval  platforms: 

1 .  Aguiar,  S.,  &  Monte,  P.  (201 1 )  Virtual  worlds  for  C2  design,  analysis,  and  experimentation. 
Paper  presented  at  the  16th  ICCRTS  conference,  Quebec.  Naval  Undersea  Warfare  Center, 
Newport,  Rl  (p.8  plus  presentation  slide  deck).  Retrieved  from:  http://www.dtic.mil/cqi- 
bin/GeiTRDoc?AD=ADA5471 57 

2.  Boldyreff,  C.,  Dastbaz,  M.,  Liu,  H.,  &  Arafa,  Y.  (2011).  Engineering  Advanced  Training 
Environment  for  Crisis  Management:  The  Pandora  Project.  Proceedings  of  Advances  in 
Computing  and  Technology  Conference,  USA,  pp.  125-131.  Retrieved  from: 

http://pandora.xlab.si/media/uploads/files/PandoraACT  1 1  .pdf 

3.  Carvalho,  M.  M.,  &  Ford,  R.  (2012).  NextVC2:  A  next  generation  virtual  world  command  and 
control.  Proceedings  of  the  Military  Communications  Conference,  USA,  pp.  1-6.  Retrieved 
from:  http://www.researchqate.net/publication/233401833  NextVC2  - 

A  Next  Generation  Virtual  World  Command  and  Control/file/d91 2f50f19f20c9f24.pdf 

4.  Champsas,  I.,  Leftheris,  I.,  Tsiatsos,  T.,  Terzidou,  T.,  &  Mavridis,  A.  (2012).  OpenGames:  A 
Framework  for  Implementing  3D  Collaborative  Educational  Games  in  OpenSim. 

Proceedings  of  the  6th  European  Conference  on  Games  Based  Learning,  Academic 
Conferences  Limited,  p.  82. 

5.  Cohen,  D.,  Sevdalis,  N.,  Patel,  V.,  Taylor,  M.,  Lee,  H.,  Vokes,  M.,  &  Darzi,  A.  (2013). 

Tactical  and  operational  response  to  major  incidents:  feasibility  and  reliability  of  skills 
assessment  using  novel  virtual  environments.  Resuscitation. 

6.  Dafli,  E.  L.,  Vegoudakis,  K.  I.,  Pappas,  C.,  &  Bamidis,  P.  D.  (2009).  Re-use  and  exchange  of 
an  Opensim  platform  based  learning  environment  among  different  medical  specialties  for 
clinical  scenarios.  Proceedings  of  Information  Technology  and  Applications  in  Biomedicine, 
9th  International  Conference,  pp.  1-5. 

7.  Delp.,  S.  L.,  Anderson,  F.  C.,  Arnold,  A.  S.,  Loan,  P.,  Habib,  A.,  John,  C.  T.,  Guendelman,  E. 
and  Thelen,  D.  G.  (2007).  OpenSim:  Open-Source  Software  to  Create  and  Analyze  Dynamic 
Simulations  of  Movement.  IEEE  Transactions  on  Biomedical  Engineering,  54  (11),  pp.  1940- 
1950. 

8.  Diener,  S.,  Windsor,  J.,  &  Bodily,  D.  (2009).  Design  and  Development  of  Medical 
Simulations  in  Second  Life  and  OpenSim.  Educause,  Australasia. 

9.  Gardner,  M.,  &  Horan,  B.  (2011).  +SPACES:  Serious  Games  for  Role-Playing  Government 
Policies.  Proceedings  of  Researching  Learning  in  Immersive  Virtual  Environments,  The 
Open  University,  Milton  Keynes,  UK,  pp.  102-112.  Retrieved  from: 
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http://repository.essex.ac.Uk/3849/1 /%2BSPACES%20Serious%20Games%20for%20Role- 

Plavinq%20Government%20Policies.pdf 

10.  Garrett,  R.  B.,  Tolk,  A.,  &  Bacon,  T.  J.  (2009).  Exploring  effective  methods  for  modelling  a 
comprehensive  approach  to  Political,  Military,  Economic,  Social,  Information,  and 
Infrastructure  (PMESII)/Human,  Cultural,  Social,  and  Behavorial  (HSCB)  Community  of 
Interest  (Col).  Proceedings  of  the  Winter  Simulation  Conference,  IEEE,  pp.  2860-2866. 

11.  Hudson,  K.,  &  Nissen,  M.  E.  (2010).  Command  &  Control  in  Virtual  Environments:  Designing 
a  Virtual  Environment  for  Experimentation.  Naval  postgraduate  school,  Montery,  CA. 

12.  Koutsabasis,  P.,  Vosinakis,  S.,  Malisova,  K.,  and  Paparounas,  N.  (2011).  On  the  value  of 
virtual  worlds  for  collaborative  design.  Retrieved  from: 

http://extev.syros.aeqean.gr/papers/J55.pdf 

13.  Krause,  D.,  Strickler,  M.,  &  Clark-Casey,  J.  (2010).  Utilizing  Open  Source  Virtual  World 
Platforms  for  Business  and  Serious  Games.  Journal  of  Competence  Management  for  Open 
Innovation:  Tools  and  It  Support  to  Unlock  the  Innovation  Potential  Beyond  Company 
Boundaries,  30,  p.  299. 

14.  Liu,  H.  and  Duffy,  A.  M.  (2012).  Enabling  behaviour  reuse  in  development  of  virtual 
environment  applications.  Proceedings  of  the  2012  Winter  Simulation  Conference.  Retrieved 
from:  http://www.simulation.kiev.ua/doc%5Cconference%5Cwsc%5C2012%5Cinv186.pdf 

15.  Liu,  H.,  Arafa,  Y.,  Boldyreff,  C.,  &  Dastbaz,  M.  (2011)  Cost-effective  virtual  world 
development  for  serious  games.  Proceedings  of  Games  Innovation  Conference,  IEEE,  pp. 
48-51. 

16.  Mavridis,  A.,  Konstantinidis,  A.,  &  Tsiatsos,  T.  (2012).  A  Comparison  of  3D  Collaborative 
Virtual  Learning  Environments:  OpenSim  vs.  Second  Life.  International  Journal  of  e- 
Collaboration  (IJeC),  8(4),  pp.  8-21.  Retrieved  from: 

http://www.fardinpour.info/svstem/files/myfiles/Mavridis  etc%282012%29ComparisonVLE 

OpenSim  SecondLife.pdf 

17.  Michel,  M.  K.,  Helmick,  N.  P.,  &  Mayron,  L.  M.  (2011).  Cognitive  cyber  situational  awareness 
using  virtual  worlds.  Proceedings  of  Cognitive  Methods  in  Situation  Awareness  and  Decision 
Support  (CogSIMA),  First  International  Multi-Disciplinary  Conference,  IEEE,  pp.  179-182. 

18.  Miehling,  J.,  Kruger,  D.,  &  Wartzack,  S.  (2013).  Simulation  in  Human-Centered  Design- 
Past,  Present  and  Tomorrow.  In  Smart  Product  Engineering  (pp.  643-652).  Springer  Berlin 
Heidelberg.  Retrieved  from:  http://link.springer.eom/chapter/1 0. 1 007/978-3-642-30817-8  63 

19.  Muller,  D.  and  Schaf,  F.  M.  (2009).  A  low  cost  learning  environment  for  collaborative 
engineering.  Retrieved  from:  http://coral.ufsm.br/eduworlds/public/papers/schaf  rev2009.pdf 

20.  Mumme,  C.,  Olivier,  H.,  &  Pinkwart,  N.  (2008).  A  Framework  for  Interaction  Analysis  and 
Feedback  in  Collaborative  Virtual  Worlds.  Proceedings  of  the  14th  International  Conference 
on  Concurrent  Enterprising,  pp.  143-150.  Retrieved  from:  http://www.ve- 
forum.orq/proiects/408/ICE%202008/Collaborative%20Systems/016.3%20- 
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%20CWE23%20Mumme  Olivier  Pinkwart  Framework  for  CVE  analysis%20  final%20SS 

6  2.pdf 

21.  Nakasone,  A.,  Prendinger,  H.,  Miska,  M.,  Lindner,  M.,  Horiguchi,  R.,  &  Kuwahara,  M.  (2011). 
Openenergysim:  A  3d  internet  based  experimental  framework  for  integrating  traffic 
simulation  and  multi-user  immersive  driving.  Proceedings  of  the  4th  International  ICST 
Conference  on  Simulation  Tools  and  Techniques,  pp.  490-498.  Retrieved  from:  http://i- 
transportlab.ip/publications/papers/pdf/SimWorks201 1  Helmut.pdf 

22.  Onyesolu,  M.  (2009).  Virtual  Reality  Laboratories:  An  ideal  solution  to  the  problems  facing 
laboratory  setup  and  management.  Proceedings  of  the  World  Congress  on  Engineering  and 
computer  science.  Retrieved  from: 

http://www.iaeng.org/publication/WCECS2009/WCECS2009  pp291-295.pdf 

23.  Pierera,  I.,  Allision,  C.,  and  Miller,  A.  (2010).  A  Use  Case  Analysis  for  Learning  in  3D  MUVE: 
A  Model  Based  on  Key  eLearning  Activities.  Retrieved  from: 

http://www.icvl.eu/2010/disc/icvl/documente/pdf/met/ICVL  ModelsAndMethodologies  paper 

15.pdf 

24.  Scacchi,  W.,  Brown,  C.,  &  Nies,  K.  (2012,  June).  Exploring  the  potential  of  computer  games 
for  decentralized  command  and  control.  Proceedings  of  the  17th  International  Command  & 
Control  Research  &  Technology  Symposium  (ICCRTS).  Retrieved  from: 
http://dodccrp.org/events/17th  iccrts  2012/post  conference/papers/1 04.pdf 

25.  Intel  Corporation  (2009).  Science  Sim:  A  virtual  environment  for  collaborative  visualization 
and  experimentation.  White  Paper.  Retrieved  from: 

http://software.intel.eom/sites/default/files/m/d/4/1/d/8/ScienceSimWP  CCEFinal.pdf 

26.  von  Kapri,  A.,  Ullrich,  S.,  Brandherm,  B.,  &  Prendinger,  H.  (2009).  Global  lab:  an  interaction, 
simulation,  and  experimentation  platform  based  on”  second  life”  and”  opensimulator”. 
Proceedings  of  the  Pacific-Rim  Symposium  on  Image  and  Video  Technology.  Retrieved 
from:  http://anettevonkapri.org/wp-content/uploads/201 2/1 2/KUB+2009.pdf 

27.  Wagner,  G.  (2010).  Model-driven  engineering  of  Second-Life-style  simulations.  Proceedings 
of  the  Winter  Simulation  Conference,  IEEE,  pp.  791-798. 

28.  Watte,  J.  (2009).  Virtual  world  interoperability:  let  use  cases  drive  design.  Journal  of  Virtual 
Worlds  Research,  2(3),  pp.  1-13.  Retrieved  from:  http://ivwr-ojs- 
utexas.tdl.org/ivwr/index.php/ivwr/article/download/727/526 

29.  Winkler,  S.  E.  (2010).  Licensing  Considerations  for  OpenSim-Based  Virtual  Worlds. 
Retrieved  from:  http://iournals.tdl.org/ivwr/index.php/ivwr/article/download/871/636 

The  list  of  references  to  review  was  further  reduced  to  focus  on  the  most  relevant  documents 
and  match  the  level  of  effort  available  in  the  contract.  The  following  10  documents  were 
reviewed  in  further  detail  to  address  the  DRDC  interests  outlined  in  Section  1.3: 

1 .  Aguiar,  S.,  &  Monte,  P.  (201 1 )  Virtual  worlds  for  C2  design,  analysis,  and  experimentation. 
Paper  presented  at  the  16th  ICCRTS  conference,  Quebec.  Naval  Undersea  Warfare  Center, 
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Newport,  Rl  (p.8  plus  presentation  slide  deck).  Retrieved  from:  http://www.dtic.mil/cgi- 
bin/GetTRDoc?AD=ADA547 1 57 


2.  Allen,  P.D.  &  Demcahk,  C.  C.  (2011).  Applied  virtual  environments:  applications  of  virtual 
environments  to  government,  military,  and  Business  Organizations.  Journal  of  Virtual  Worlds 
Research,  4(1),  pp.  1-27. 

3.  Carvalho,  M.  M.,  &  Ford,  R.  (2012).  NextVC2:  A  next  generation  virtual  world  command  and 
control.  Proceedings  of  the  Military  Communications  Conference,  USA,  pp.  1-6.  Retrieved 
from:  http://www.researchgate.net/publication/233401833  NextVC2  - 

A  Next  Generation  Virtual  World  Command  and  Control/file/d91 2f50f19f20c9f24.pdf 

4.  Hsu,  E.  B.,  Li,  Y.,  Bayram,  J.  D.,  Levinson,  D.,  Yang,  S.,  &  Monahan,  C.  (2013).  State  of 
Virtual  Reality  Based  Disaster  Preparedness  and  Response  Training.  PLoS  currents,  5. 
Retrieved  from:  http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3644293/ 

5.  Maxwell,  D.  (2009).  Application  of  Virtual  World  Technologies  to  Undersea  Warfare 
Learning.  Naval  Undersea  Warfare  Center  Division,  Newport,  Rl.  Retrieved  from: 

http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA530269 

6.  Maxwell,  D.,  Aguiar,  S.,  Monte,  P.,  &  Nolan,  D.  (2011).  Two  Navy  Virtual  World 
Collaboration  Applications:  Rapid  Prototyping  and  Concept  of  Operations  Experimentation. 
Journal  of  Virtual  Worlds  Research,  4(2).  Retrieved  from:  http://jvwr-ois- 
utexas.tdl.orq/ivwr/index.php/ivwr/article/download/21 1 3/5551 

7.  Scacchi,  W.,  Brown,  C.,  &  Nies,  K.  (2012,  June).  Exploring  the  potential  of  computer  games 
for  decentralized  command  and  control.  Proceedings  of  the  17th  International  Command  & 
Control  Research  &  Technology  Symposium  (ICCRTS).  Retrieved  from: 
http://dodccrp.org/events/17th  iccrts  2012/post  conference/papers/1 04.pdf 

8.  Smith,  R.  (201 1 )  Creating  a  killer  app  for  the  department  of  defence.  Journal  of  Virtual 
Worlds  Research,  4(1),  pp.  1-5. 

9.  Tate,  A.,  Dalton,  J.,  &  Potter,  S.  (2009).  1-Room:  a  Virtual  Space  for  Emergency  Response 
for  the  Multinational  Planning  Augmentation  Team.  Proceedings  of  the  Fifth  International 
Conference  on  Knowledge  Systems  for  Coalition  Operations  (KSCO-2009)  (Vol.  31), 
Chilworth  Manor,  Southampton,  UK,  March,  pp.  1  -5. 

10.  von  Kapri,  A.,  Ullrich,  S.,  Brandherm,  B.,  &  Prendinger,  H.  (2009).  Global  lab:  an  interaction, 
simulation,  and  experimentation  platform  based  on”  second  life”  and”  opensimulator”. 
Proceedings  of  the  Pacific-Rim  Symposium  on  Image  and  Video  Technology.  Retrieved 
from:  http://anettevonkapri.org/wp-content/uploads/201 2/1 2/KUB+2009.pdf 
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3  RESULTS 

This  section  presents  the  results  of  the  literature  survey  including  examples  of  vWorld  platforms, 
identified  users  and  applications  of  the  OpenSim  platform  in  the  areas  of  interest  to  DRDC. 

3.1  vWorld  Platforms 

Examples  of  vWorld  platforms  extracted  from  the  surveyed  literature  are  presented  in  this 
section.  Identified  users  and  applications  of  the  OpenSim  platform  from  the  defence  and 
security  realm  are  noted  accordingly  in  Section  3.2.  Most  of  the  simulation  environments  allow 
the  user  to  assemble  predefined  objects  in  various  ways.  Content  creation  is  done  outside  of  the 
vWorlds  using  modelling  tools  (e.g.  AutoCAD,  SketchUp,  and  Autodesk  Maya)  that  support 
common  industry  object  formats  (e.g.  Collaborative  Design  Activity,  COLLADA). 

3.1.1  Second  Life 

Second  Life  (SL,  http://secondlife.com)  is  a  commercial,  persistent,  3-Dimensional  (3D),  virtual 
world  that  provides  simple  modelling  tools  and  a  scripting  language  for  the  creation  of 
interactive  objects  and  the  alteration  of  the  worlds’  physics.  In  SL,  educators  can  create 
interactive  activities  for  the  support  of  experiential,  project-based  and  community  service-based 
learning  (Bedford,  Birkedal,  Erhard,  Graff,  &  Hempel,  2006).  In  SL,  chatting  with  others  and 
navigating  the  vWorld  is  without  charge,  but  creating  permanent  objects  and  buildings  requires 
the  acquisition  of  virtual  land  from  the  developers  or  from  other  users. 

3.1.2  OpenSIM 

OpenSIM  (http://opensimulator.org)  is  an  open  source  server  platform  for  hosting  vWorlds.  Its 
main  feature  is  the  compatibility  with  the  SL  client  using  the  SL  protocol  for  client-to-server 
communication.  The  modelling  tools  and  the  scripting  language  are  similar  to  those  used  in  SL 
(Gardner  &  Horan,  2011;  Mavridis,  Konstantinidis,  &Tsiatsos,  2012).  Creating  objects  and 
importing  things  (e.g.,  textures)  in  OpenSIM  is  without  charge.  Every  user  has  private  land  on 
which  they  can  create  their  own  environment.  Gaining  access  to  an  OpenSIM  public 
environment  can  only  be  done  once  the  administrator  of  that  particular  OpenSIM  virtual  world 
grants  the  user  permission  to  access  that  world  (Pierera,  Allison,  &  Miller,  2010);  however,  the 
OpenSIM  software  may  be  downloaded  and  used  free  of  charge  on  private  networks. 

3.1.3  VBS2/VBS3 

Virtual  Battlespace  3  (VBS3;  http://products.bisimulations.com/),  and  its  predecessor  VBS2,  is  a 
serious-game-based  virtual  environment,  providing  terrain  and  scenario  editors.  VBS2  has 
focused  primarily  on  land  combat  although  other  domains  are  possible.  VBS3  provides  a 
development  user  interface  to  modify  and  extend  the  capability  inherent  in  the  product.  Note 
that  the  persistence  capability  inherent  in  many  vWorld  products  is  missing  in  VBS3.  DND  has 
approved  VBS2  for  use  on  the  DWAN  and  it  is  currently  being  applied  by  the  Director  Land 
Synthetic  Environments  (DLSE)  Canadian  Forces  Base  (CFB)  Kingston. 
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3.1.4  Active  Worlds 

In  Active  Worlds  (http://www.activeworlds.com),  users  communicate  with  others  and  create  their 
own  personal  3D  virtual  space.  The  educational  capabilities  of  Active  Worlds  have  been 
explored  by  a  learning  community  known  as  Active  Worlds  Educational  Universe  (AWEDU)  that 
provides  educators,  students,  educational  institutions  and  individual  programs  with  the  Active 
Worlds  technology  (Dalfi,  Vegoudakis,  Pappas,  &  Bamidis,  2009).  Through  Active  Worlds, 
educators  can  assess  new  ideas,  learning  theories  and  teaching.  Institutions  already  using 
Active  Worlds  include  the  Boston  Museum  of  Science,  the  San  Jose  Tech  Museum  of 
Innovation,  the  National  Aeronautics  and  Space  Administration  (NASA)  Ames  Research 
Laboratory  (Mavridis,  Konstantinidis,  &Tsiatsos,  2012). 

3.1.5  Croquet 

Croquet  (http://opencroquet.org)  is  an  open  source  cross  platform  3D  environment  designed  for 
multi-user  interaction  and  simulation  (McCahill,  2004).  Written  in  Squeak,  an  object  and  class 
based,  reflective  Smalltalk  implementation,  Croquet  is  a  combination  of  open  source  software 
and  peer-to-peer  network  architecture  providing  an  infrastructure  for  synchronous,  real-time 
problem  solving  within  shared  simulations  (Kadobayashi,  Lombardi,  McCahill,  Stearns,  Tanaka, 
&  Kay,  2005;  Watte,  2009;  Mavridis,  Konstantinidis,  &  Tsiatsos,  2012). 

3.1.6  OLIVE 

Science  Applications  International  Corporation’s  (SAIC)  On-Line  Interactive  Virtual  Environment 
(OLIVE3)  3.0  is  a  cloud  based  platform  that  runs  on  Virtual  Machine  based  servers  that  are 
always-on.  The  functionality  of  OLIVE  allows  for  basic  communication  and  person-to-person 
interaction  as  in  both  SL  and  Teleplace. 

3.1.7  Open  Cobalt  Alpha 

Open  Cobalt  Alpha  (http://www.opencobalt.org/)  is  an  open  source  platform  that  uses  3D 
multimedia  wiki  technology  that  allows  the  user  to  create  collaborative  and  hyperlinked  multi¬ 
user  virtual  workspaces  on  major  software  operating  systems.  Cobalt  Alpha  vWorlds  may  be 
hyperlinked  via  3D  portals  to  form  a  distributed  network  of  interconnected  collaboration  spaces 
using  a  peer-based  messaging  protocol  that  reduces  demands  on  server  infrastructures 
(Gardner  &  Horan,  201 1 ).  It  supports  integrated  web  browsing,  voice  chat,  text  chat  as  well  as 
access  to  remote  desktop  applications  and  services. 

3.1.8  OpenEnergySim 

OpenEnergySim  is  an  integrative  vWorlds  based  platform  that  combines  traffic  simulation  and 
user  driving  experience  with  the  goal  of  providing  traffic  engineers  with  a  simulation  tool  for 
exploring  and  testing  Intelligent  Transportation  System  (ITS)  strategies  (Nakosone  et  al.,  2011). 


3  Highlights  of  OLIVE  3.0.  Retrieved  on  03  April  2014  from: 

https://www.leidos.com/sites/default/files/imported/products/simulation/olive/download/olive-hiqhliqhts.pdf 


24  June  2014 


10 


5457-002  Version  02 


CAE 

vWorld  Capability  Development  Support 
Literature  Survey 


3.1.9  Open  Wonderland 

Open  Wonderland  (http://openwonderland.org)  is  a  Java-based  open-source  toolkit  for  creating 
collaborative  3D  vWorlds.  Basic  features  allow  users  to  interface  with  external  applications 
(e.g.,  Open  Office,  Firefox,  and  Skype)  visualized  as  2D  windows  in  the  vWorld  and  to  add 
audio/video  recordings  or  streaming  audio/video  to  in-world  objects  (Koutsabasis,  Vosinakis, 
Malisova,  &  Paparounas,  2011).  The  Open  Wonderland  Foundation  operates  two  community 
servers  that  are  intended  for  user  created  vWorlds  for  demonstration  purposes  only.  The 
servers  are  reset  on  a  regular  basis,  so  individual  work  cannot  be  saved  on  these  servers.  Open 
Wonderland  also  offers  security  and  authentication  features  for  user  groups  (Mavridis, 
Konstantinidis,  &  Tsiatsos,  2012).  Instructions  for  connecting  to  a  community  server  can  be 
found  on  the  Open  Wonderland  website. 

3.1.10  Pivote 

Pivote  (http://www.pivote.info)  is  an  open  source  virtual  authoring  system  for  vWorlds  orignially 
developed  for  training  medical  professionals  in  higher  education.  All  the  structure  and 
information  content  of  an  exercise  is  stored  on  the  web,  not  in  the  vWorld,  permitting 
development  of  courseware  independent  of  the  vWorld  to  facilitate  reuse  of  models  and 
courseware  across  a  number  of  applications  (Gardner  &  Horan,  2011;  Mavridis,  Konstantinidis, 
&  Tsiatsos,  2012). 

3.1.11  Teleplace 

Teleplace  (http://telexlr8.wordpress.com/openqwag/teleplace/)  is  a  3D  videoconferencing 
environment  that  offers  basic  functionality  to  support  telework,  online  meetings,  group 
collaboration  and  e-learning.  The  spaces  within  Teleplace  are  interior  only  spaces,  with  no 
access  to  natural  landscapes.  Hudson  and  Nissen  (2010)  state  that  Teleplace  does  not  offer 
any  JavaScript  functionality;  nor  does  it  provide  the  source  code.  The  authors  also  state  that 
Python  scripting  language  is  supported  by  Teleplace,  which  would  allow  developers  to  recreate 
the  scripted  content.  However,  significant  reworking  of  these  models  may  be  required  to  make 
even  minor  changes  to  the  experimental  spaces. 

3.2  Overview  of  Documents 

This  section  provides  a  brief  overview  of  the  documents  reviewed  in  accordance  to  the  revised 
scope  of  objectives.  Additional  details  regarding  the  applications  and  user  community  of 
OpenSIM  are  presented  in  Section  3.3.  Issues  and  requirements  for  systems  development  and 
training  applications  are  presented  in  Section  4.1  and  Section  4.2,  respectively. 

3.2.1  Aguiar  &  Monte  (2011) 

This  paper  describes  how  vWorlds  can  enhance  the  design  process  for  submarine  C2  systems 
by  involving  fleet  personnel  in  design  and  experimentation.  The  US  Navy  has  investigated  and 
applied  multiple  applications  of  vWorlds  technology  to  the  undersea  warfare  (USW)  domain  for 
Anti-Submarine  Warfare  (ASW)  and  mine  warfare.  One  area  of  application  that  is  being 
supported  is  C2  collaborative  engineering  and  concept  of  operation  experiments  (COOPEXs). 
Collaborative  engineering  is  a  systematic  approach  to  the  integrated,  concurrent  design  of  C2 
systems  ranging  from  a  single  system  (e.g.,  ship  navigation)  to  a  full,  multisystem  Command 
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Center.  The  COOPEX  involves  experimenting  with  proposed  C2  concepts  in  a  vWorld,  before 
committing  the  operational  fleet  to  new  equipment,  operating  procedures,  or  manning  concepts. 
The  authors  describe  how  vWorlds  can  offer  an  environment  for  multidisciplinary  groups  of 
stakeholders  (i.e.  Integrated  Product  Teams,  IPT)  to  effectively  design  prototype  C2  systems  in 
real  time  while  utilizing  video,  voice,  chat  and  real-time  modelling  tools  for  modifying  the  virtual 
environment.  For  example,  in  October  2010  a  program  under  the  Office  of  Naval  Research  held 
a  week-long  arrangement  studies  workshop  in  Groton,  CT,  during  which  submarine  crews,  with 
the  aid  of  C2  subject  matter  experts  and  cognitive  scientists,  generated  10  separate  Command 
and  Control  Center  (CACC)  arrangements  in  a  vWorld  in  real-time.  These  prototypes  were  then 
used  to  analyze  pre-recorded  C2  events  to  visually  expose  each  architecture  component  (e.g. 
workspace,  human  communications,  Human-System  Interface  (HSI),  team  structure,  work  flow, 
task  flow,  automation  or  training)  as  it  affects  a  specific  mission  scenario.  In  addition,  vWorlds 
allowed  these  virtual  prototypes  to  be  evaluated  by  fleet  personnel  immersed  into  the 
environment  with  enough  fidelity  to  allow  operator  performance  prediction. 

3.2.2  Allen  &  Demchak  (2011) 

This  paper  presents  a  set  of  vWorlds  features  and  their  use  in  a  wide  range  of  application  areas 
involving  training,  operational,  and  rehearsal  objectives. 

The  authors  propose  four  main  advantages  of  vWorlds  compared  with  conventional  design 
methods,  although  the  emphasis  on  the  need  for  a  high  degree  of  realism  is  contentious  and 
unsupported  by  the  scientific  literature  in  general  (Lintern  et  al.,  1997;  Salas  et  al.,  2009; 
Smallman  &  St.John,  2005).  The  four  noted  advantages  are: 

•  Provide  as  close  of  a  representation  of  reality  as  possible; 

•  Allow  selected  laws  of  physics  to  be  violated.  Enhanced  reality  involves  providing  additional 
information  to  the  participant  that  would  not  be  available  in  the  real  world,  but  facilitates  the 
achievement  of  the  application’s  objectives.  For  instance,  the  ability  to  see  the  layout  and 
functioning  of  a  ship’s  sub-systems  may  be  essential  to  achieving  the  training  objectives  for 
the  maintenance  of  those  systems; 

•  Provide  different  levels  of  resolution  to  demonstrate  how  different  echelons  of  management 
view  the  same  situation.  For  example,  a  ship  will  have  a  detailed  view  of  its  area  of 
responsibility  but  the  Maritime  Component  Command  element  ashore  will  have  a  view  of  the 
ship  that  is  much  reduced  in  detail.  The  ability  not  only  to  see  these  different  echelons  and 
perceptions  but  also  to  be  able  to  quickly  shift  from  one  to  the  next,  may  be  advantageous  to 
the  participants  depending  on  the  application  areas;  and, 

•  Provide  perceived  realities  based  on  the  perceptions  of  various  participants.  Camera  options 
can  permit  users  in  vWorlds  to  represent  how  each  friendly,  enemy,  neutral  or  wavering 
entity  views  each  other’s  situation.  This  ability  is  useful  for  training  and  for  purposes  such  as 
wargaming  where  Blue  forces  take  the  role  of  the  adversary  to  identify  weaknesses  in 
potential  courses  of  action.  Seeing  what  you  can  see  and  seeing  what  you  think  others  can 
see  can  be  essential  to  planning  and  other  Information  Operations. 
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3.2.3  Carvalho  &  Ford  (2012) 

This  paper  presents  a  vWorld-based  approach  to  C2  for  joint  operations  where  coalition 
partners  participate  in  a  common  vWorld  C2  environment  (e.g.,  virtual  ops  room,  bridge,  forward 
operating  base,  and  vehicle  on  patrol).  A  prototype  of  the  proposed  capability  was  implemented 
and  demonstrated  as  an  extension  to  OpenSIM.  The  authors  describe  the  approach  but  provide 
very  limited  details  of  the  prototype  implementation  and  preliminary  capability  demonstrations. 

Much  of  the  paper  is  dedicated  to  discussing  the  differential  presentation  of  information  to  users 
with  varying  security  restrictions.  A  common  approach  for  information  protection  is  to  create  a 
physical  environment  whereby  multiple  vWorlds  exist  and  information  can  be  shared,  tightly 
constraining  access  to  such  environments  only  to  authorized  parties.  They  cite  a  number  of 
studies  of  C2  systems  in  virtual  reality  settings  but  provide  no  summary  of  the  findings;  they 
note  that  the  focus  has  been  on  the  user  interaction  with  the  simulation  and  collaboration 
capabilities  of  the  virtual  environments  but  provide  no  evidence  from  the  studies  to  suggest  that 
the  vWorlds  were  valid  for  those  applications. 

The  authors  believe  that  being  able  to  selectively  and  automatically  control  the  release  of 
information  to  users  is  critical  to  widespread  adoption  of  vWorlds  for  C2  training  and  use.  They 
approach  the  “context-dependent  information  release”  problem  by  controlling  access  to 
information  through  administrative  computer  policies  that  seem  to  hide  the  details,  in  effect, 
preventing  the  unauthorized  user  from  knowing  what  they  don’t  know. 

The  authors  recommend  performing  a  requirements  analysis  for  selective  access  to  sensitive 
resources  or  information  in  vWorld  applications.  Much  of  this  information  should  be  represented 
in  policies,  doctrine,  standard  operating  procedures,  tactics,  techniques  and  procedures,  and 
regulatory  constraints,  but  it  is  often  up  to  humans  in  the  environment  to  enforce  such 
constraints  during  ad  hoc  meetings  and  discussions  which  may  be  impractical  in  time  critical 
settings. 

3.2.4  Hsu  etal.  (2013) 

This  paper  is  a  literature  review  that  lists  applications  of  vWorld-based  training  for  disaster 
preparedness  and  response  in  the  United  States  by  a  variety  of  government  departments  and 
agencies  including  the  U.S.  Department  of  Homeland  Security  (DHS),  the  Centers  for  Disease 
Control  and  Prevention  (CDC)  as  well  as  academic  institutions.  The  authors  are  interested  in 
determining  whether  virtual  reality  based  simulation  technologies  (of  which  vWorlds  are  but  one 
class  of  simulation  approaches)  can  augment  or  improve  on  more  traditional  approaches  to 
training,  particularly  for  events  that  involve  sizable  participants  that  must  interact  both  with  each 
other  and  with  the  environment.  They  note  that  a  number  of  academic  institutions  have 
evaluated  virtual  reality  training  effectiveness  in  some  applications,  but  they  cite  no  publications. 
They  do  go  on  to  cite  several  government  and  academic  studies  of  virtual  reality  training 
simulations,  but  they  do  not  summarize  the  findings  from  these  studies. 

They  note,  however,  that  there  are  some  caveats  to  vWorld  training  including  lack  of  support 
from  training  managers,  presumably  due  to  the  lack  of  evidence  that  the  technology  is  effective, 
and  they  cite  the  need  for  validation  studies  to  support  cost-effectiveness  arguments.  They  also 
note  that  these  applications  are  still  simulations  and  that  user  interaction  with  the  simulation  can 
differ  markedly  from  the  real  life  counterpart,  leading  to  additional  training  requirements  simply 
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to  use  the  training  device  and  frequently  not  providing  proprioceptive  or  tactile  feedback  thought 
to  be  necessary  in  some  applications. 

The  paper  includes  a  list  of  academic  programmes  and  commercial  ventures  developing  virtual 
reality  based  environments  aimed  at  disaster  and  medical  preparedness  training.  Unfortunately, 
they  focus  on  the  functionality  of  these  instantiations  rather  than  what  they  are  effective  at 
accomplishing,  making  no  statement  on  whether  the  applications  are  valid  or  if  validation 
studies  have  been  conducted. 

Examples  of  vWorld  applications  noted  in  this  paper  include: 

•  The  use  of  a  virtual  reality  patient  simulation  system  for  teaching  emergency  response  skills 
to  U.S.  Navy  medical  providers  (Freeman  et  al.,  2001).  Although  this  reference  pre-dates 
OpenSIM,  it  may  be  an  example  of  a  vWorld  domain  that  could  be  explored  in  OpenSIM; 

•  Immersive  simulation  for  training  first  responders  for  mass  casualty  incidents  (Wilkerson  et 
al.,  2008); 

•  Simulation  for  team  training  and  assessment  using  virtual  worlds  (Becerra-Fernandez  et  al., 
2008); 

•  Virtual  simulation  as  an  instructional  method  for  nursing  students  was  demonstrated  to 
reinforce  learning  and  improve  learning  retention  over  time  (Farr  et  al.,  2013); 

•  New  York  City  Office  of  Emergency  Management  (OEM)  is  using  the  Advanced  Disaster 
Management  Simulator  (ADMS4)  training  system  developed  by  Environmental  Tectonics 
Corporation  that  employs  artificial  intelligence.  The  system  focuses  on  command  element 
simulation  and  allows  trainees  to  navigate  through  the  virtual  replica  city  by  means  of  a 
joystick.  Trainees  can  communicate  emergency  response  needs  through  a  facilitator  who 
guides  them  through  decision  points  and  objectives;  and, 

•  The  Los  Angeles  Police  Department  is  using  the  commercially  available  Hydra  Simulation 
Training  System5  to  foster  disaster-based  training  for  its  incident  command  officers.  This 
system  can  be  interlinked  with  other  emergency  operation  centers  around  the  world.  With  a 
variety  of  newscasts,  briefings  and  other  simulated  real-time  information,  Hydra  features 
immersive  simulation  training  with  video  feeds  that  monitor  decision-making  processes. 

3.2.5  Maxwell  etal.  (2011) 

This  paper  describes  the  Naval  Undersea  Warfare  Center  (NUWC)  work  with  vWorlds  for 
applications  including  rapid  prototyping  and  design  activities,  virtual  concept  of  operations 
exercises  and  training  Target  Motion  Analysis  (TMA).  They  note  that  traditional  design 
evaluation  methods  involving  scale  models  and  full  size  mock  ups  are  useful  but  can  be 
expensive  and  they  are  typically  non-functional.  NUWC  is  exploring  the  use  of  vWorlds  to 
provide  a  virtual  mock  up  environment  to  simulate  submarine  work  spaces  that  attempt  to 


4  Available  at  http://www.traininqfordisastermanaqement.com.  Accessed  on  07  January  2014. 

5  Hydra  System  Simulation  Training.  Available  at:  http://www.govtech.com/public-safety/Los-Angeles-Police- 
Hydra-Simulation-Training.html.  Accessed  on  07  January  2014. 
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constrain  and  enable  the  user  similar  to  the  real  (or  envisioned)  submarine  to  assess  design 
and  information  flow.  They  note  that  a  balance  must  be  struck  between  security  and  connectivity 
to  allow  remote  users  to  participate  in  the  design  evaluation,  but  also  note  that  there  are  several 
secure  networking  technologies  available.  They  also  recognize  that  the  vWorld  environment 
needs  to  support  a  number  of  human  factors  elements  such  as  natural  communications,  sight 
lines  and  physical  restrictions  if  the  environment  is  to  be  used  to  assess  usability. 

The  requirements  and  challenges  for  vWorld  selection  of  these  applications  are  provided. 
Modification  of  the  vWorld  contents  should  be  possible  while  the  simulation  is  running  so  that 
designs  can  evolve  effectively.  They  also  note  that  users  will  come  from  a  wide  range  of 
backgrounds  so  that  the  user  interface  should  be  designed  for  both  the  naive  and  expert  user. 
The  authors  describe  the  design,  test  and  evaluation  of  a  Command  and  Control  Center 
(CACC)  inside  a  vWorld  simulation  of  operations  exercises.  By  transforming  physical  mock  ups 
into  virtual  objects,  the  costs  of  rapid  prototyping  are  demonstrated  to  be  drastically  reduced. 
They  noted  that,  at  the  time,  Second  Life  was  the  only  vWorld  environment  that  had  a  user 
interface  and  model  modification  capability  adequate  to  meet  their  rapid  prototyping 
requirements,  although  other  environments  actually  provided  better  support  for  importing  wire- 
mesh  models.  A  hypothetical  Virginia  class  attack  centre  was  created  in  Second  Life  with 
sufficient  detail  to  allow  users  to  perform  certain  procedures,  interacting  with  consoles  and 
viewing  information  on  screens,  in  order  to  assess  information  flow,  although  no  assessment  of 
the  effectiveness  was  reported. 

The  authors  explored  vWorld  technologies  as  a  replacement  for  Concept  of  Operations  Exercise 
(COOPEX)  that  required  different  capabilities  from  rapid  prototyping  (although  specific  details 
beyond  integrating  video  feeds  were  not  reported)  and  determined  that  the  Teleplace 
environment  was  the  most  capable  to  create  their  virtual  submarine  environment.  They 
conducted  a  study  in  the  Teleplace  virtual  submarine  as  well  as  with  real  equipment  in  a 
physical  environment,  using  both  a  naive  and  an  expert  team  of  submariners  conducting  a 
target  motion  analysis  task.  Subjective  observations  and  subject  feedback  suggested  that  both 
simulated  and  physical  environments  were  equally  effective,  but  there  were  insufficient 
performance  data  to  make  firm  conclusions.  The  authors  noted  that  the  vWorld  developer 
interface  was  awkward  and  that  “camera”  control  was  important  to  collaboration  (user  controlled 
sight  lines.)  A  major  hurdle  was  the  level  of  detail  of  imported  models,  many  of  which  had  to  be 
simplified  in  order  to  obtain  adequate  simulation  performance. 

3.2.6  Maxwell  (2009) 

This  is  a  Microsoft  PowerPoint  presentation  that  is  related  to  the  work  of  Maxwell  et  al.  (201 1 ). 
The  presentation  is  an  information  brief  that  describes  the  characteristics  of  vWorlds 
technologies  and  NUWC  mission  objectives  with  respect  to  collaboration  and  innovation  in  FY08 
(investigation  of  various  vWorlds  platforms)  and  FY09  (experimentation  to  effectively  apply 
vWorlds  in  support  of  undersea  warfare  mission  areas).  The  presentation  notes  that  NUWC  is 
organizing  a  coordinated  military  coalition  presence  (i.e.,  Army,  Navy,  Air  Force,  Marines  and 
others)  into  Second  Life.  Project  Bluejacket  is  presented  as  an  exploration  of  a  vWorld 
technology  to  teach  basic  submarine  tactical  skills  such  as  TMA. 
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3.2.7  Scacchi,  Brown  &  Nies  (2012) 

This  report  describes  results  from  a  one  year  long  research  study  investigating  how  vWorlds 
concepts,  techniques  and  tools  can  be  used  to  create  an  online  environment  that  supports 
experiments  in  Decentralized  Command  and  Control  (DCC).  The  objective  is  to  allow  remote 
users  to  work  or  train  in  virtual  command  and  control  centres  as  if  they  were  co-located  in  a 
physical  centre.  The  intent  is  to  follow  the  government  wide  initiative  to  exploit  open  source 
software  wherever  it  is  cost  effective.  Their  efforts  depart  from  current  C2  practice  but  seek  to 
advance  scientific  and  technical  knowledge  of  how  vWorlds  might  be  leveraged  to  support 
future  C2  practice. 

The  authors  describe  the  development  of  a  decentralized  prototype  computer  game  and  virtual 
world  collectively  referred  to  as  the  DECENT  system.  The  system  is  implemented  in  OpenSIM 
and  allows  authors  to  rapidly  create  a  functional  C2  vWorld  analog,  i.e.,  a  simulation  of  a  C2 
environment  (e.g.,  virtual  ops  room,  bridge,  forward  operating  base,  and  vehicle  on  patrol)  that 
works  similarly  to  the  real  C2  environment,  and  populate  it  with  users  for  concept  prototyping, 
testing  and  experimentation. 

The  authors  began  the  development  of  physical  places  in  vWorlds  by  first  inspecting  the 
components  and  layouts  of  real  C2  structures  from  images  of  C2  facilities  (e.g.,  Hanscom  Air 
Force  Base)  in  order  to  identify  elements  to  replicate  in  the  vWorlds.  The  authors  replicated 
elements  common  to  C2  facilities  including  shared  resources  such  as  large  screen  displays  and 
tabletop  surfaces  used  to  display  streaming  video,  as  well  as  static  images  and  a  mission 
planning  counterterrorism  training  game.  Other  elements  replicated  in  the  vWorld  include  private 
resources  such  as  operator  workstations  and  other  mission  resources. 

The  authors  state  that  research  is  needed  to  evaluate  and  compare  the  validity  of  alternative 
vWorlds  and  game  technologies,  and  the  affordances  they  provide,  within  a  controlled 
experimental  environment.  Replicating  real  C2  components  in  a  vWorld  provides  the  basis  for 
the  systematic  investigation  and  experimentation  of  C2  related  issues.  They  cite  experiments  to 
investigate  the  efficacy  of  team  organization,  team  dynamics,  and  training  outcomes  when  done 
in  the  real  world  in  comparison  to  a  virtual  world  (e.g.,  Bergin  et  al. ,  2010;  Hudson  &  Nissen, 
2010,  2011;  Wynn,  Ruddy,  &  Nissen,  2010),  although  summaries  of  these  efforts  are  not 
provided. 

The  authors  note  that  integrating  DECENT  with  other  currently  existing  military  OpenSIM 
projects  (e.g.,  US  Army  Military  Open  Simulator  Enterprise  Strategy,  MOSES,  2012;  NUWC 
virtual  campus,  Aguiar  &  Monte,  201 1 )  can  be  done  by  adding  a  new  region  and  importing  the 
DECENT  assets,  or  establishing  a  hypergrid  connection,  or  federation  between  the  two  virtual 
worlds. 

Overall,  the  authors  recommend  using  OpenSIM  for  vWorlds  development  because  it  was  easy 
to  use  and  allowed  them  to  quickly  develop  the  DECENT  system.  They  provide  further  reasons 
for  adopting  OpenSIM  in  comparison  to  SL  along  the  dimensions  of  cost,  users,  stability,  asset 
ownership,  and  scripting.  Based  on  these  dimensions,  OpenSIM  is  free  of  charge,  has  access  to 
smaller  groups  of  existing  users,  tends  to  have  much  more  stable  connectivity  and  voice-chat, 
offers  controlled  access  and  ownership  of  servers  and  supports  the  Linden  Scripting  Language 
and  provides  an  Application  Programming  Interface  (API)  that  additionally  allows  the  developer 
to  extend  OpenSIM  capabilities  using  C++,  Matlab  or  Python. 
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3.2.8  Smith  (2011) 

This  article  explores  the  slow  adoption  of  vWorlds  by  organizations  within  the  US  Department  of 
Defense  and  suggests  some  features  that  might  make  vWorlds  more  appealing  for  military 
training  and  game-based  applications.  Smith  notes  that  for  vWorlds  to  be  adopted  more  broadly 
by  the  Department  of  Defense,  vWorlds  must  provide  additional  capabilities  (or  at  least  more 
convenient  instantiations  of  existing  capabilities)  that  go  beyond  mere  collaboration,  because 
many  collaborative  tools  are  already  in  use.  Few  if  any  vWorlds  have  made  the  effort  to  include 
the  military  behaviours  demonstrated  in  first  person  shooter  (FPS)  virtual  environments.  Smith 
goes  on  to  state  that  developing  a  wide  variety  of  capabilities  is  beyond  any  single  company;  a 
task  that  is  distinct  from  the  creation  and  operation  of  a  vWorld  infrastructure,  although  it  seems 
a  strange  observation  when  companies  can  produce  FPS  with  such  capabilities.  Nevertheless, 
providing  tools  to  allow  developers  to  create  and  share  models  seems  to  be  a  natural  alternative 
to  a  single  company  endeavour. 

Smith  makes  an  excellent  point,  one  that  is  commonly  overlooked  in  the  current  climate  to  adopt 
all  things  technological:  “If  your  organization  were  given  (a  vWorld),  what  could  you  use  it  for  out 
of  the  box?”  Unfortunately,  this  misses  the  mark  somewhat  and  might  better  read:  “What  should 
you  use  a  vWorld  for?”  In  other  words,  where  would  it  be  cost  effective  compared  with 
alternative  approaches? 

3.2.9  Tate,  Dalton,  &  Potter  (2009) 

This  paper  describes  the  characteristics  of  an  1-Room  or  “intelligent  room”  that  serves  as  a 
knowledge  aid  to  support  collaborative  meetings.  1-Room  was  developed  to  aid  sense-making6 
about  the  current  situation  during  planning,  option  analysis,  and  decision-making.  1-Rooms  are 
being  created  on  a  number  of  virtual  world  environments  with  an  emphasis  on  sharing  concepts 
and  systems  support.  They  are  also  being  applied  to  a  range  of  collaborative  tasks  such  as 
homeland  security,  UAV  mission  monitoring,  product  design  and  review  meetings,  scientific 
project  regular  reviews  and  team  training.  The  authors  describe  the  concept  of  an  Emergency 
Response  Virtual  Collaboration  Centre,  realized  in  SL  and  OpenSIM,  within  which  it  is  possible 
to  set  up  and  maintain  a  persistent  set  of  information  assets  (e.g.  wall  posters,  maps,  etc.)  and 
dynamically  loaded  content  (such  as  imagery,  movies,  web  page  contents,  etc.)  in  an  area  that 
is  shared  by  the  participants.  The  1-Room  is  meant  to  provide  a  generic  technology  basis  for  a 
wide  range  of  potential  collaborative  applications  (see  Section  4.1 .2).  The  authors  cite  an 
example  of  a  collaborative  application,  i.e.  the  “l-X  Process  Panel”,  implemented  in  OpenSIM 
that  serves  as  a  form  of  “to-do”  list. 

3.2.10  von  Kapri  et  al.  (2009) 

This  paper  describes  a  vWorld  called  Global  Lab.  The  motivation  behind  the  work  is  to  provide  a 
framework  with  strong  support  for  collaboration  in  vWorlds  for  research  purposes.  The  Global 
Lab  platform  contains  multimodal  presentation  and  interaction  communication,  but  goes  beyond 
communication  to  demonstrate  development  and  inclusion  of  physics-based  models  of  objects 
in  the  environment.  Two  example  applications  are  presented  that  demonstrate  the  capabilities  of 
Global  Lab  for  simulation-based  experimentation  with  sensor-based  systems.  For  example,  an 


6  Sense-making  is  the  iterative  process  of  gathering  information,  interpreting  information,  and  developing  mental 
models  (conceptualizations  or  beliefs)  in  order  to  make  rational  decisions. 
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CAE 

vWorld  Capability  Development  Support 
Literature  Survey 


animation  authoring  tool  for  virtual  bots  is  described  using  an  XML-based  scripting  language 
called  Multi-modal  Presentation  Markup  Language  (MPML3D)  that  is  used  to  control  the  speech 
output  and  non-verbal  behavior  (e.g.  gestures  and  facial  expressions)  of  bots  (Artificial 
Intelligence  controlled  agents  in  simulations.)  The  MPML3D  scripting  language  can  be 
integrated  with  OpenSIM  and  SL  vWorlds  to  provide  agent  representation  for  these  domains, 
von  Kapri  et  al.  note  that  verbal  communication  in  vWorlds  is  usually  accomplished  through  a 
chat  interface  that  displays  text  on  the  screen,  although  bots  could,  in  principle,  use  a  speech 
production  interface  or  recorded  speech  to  communicate  with  virtual  users. 

3.3  OpenSIM  User  Community  and  Applications 

This  section  presents  the  user  community  of  OpenSIM  in  the  application  domains  of  interest  to 
DRDC  including:  Navy  operations,  Navy  training,  other  military  operations  and  training,  real-time 
defence  and  security  environment  (e.g.,  emergency  operations),  and  other  real-time  operational 
environments.  Also,  the  manner  that  OpenSIM  is  used  in  support  of  systems  development  is 
presented  (e.g.,  rapid  prototyping,  concept  development,  and  experimentation).  Example 
applications  are  described  and  references  are  provided  in  the  Table  3-1. 
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4  DISCUSSION 

4.1  Systems  Development 

This  section  discusses  the  potential  use  of  OpenSIM  platform  for  systems  development. 

Practical  considerations  are  proposed  to  address  some  of  the  anticipated  problems  and  missing 

requirements  for  effective  use  in  design. 

4.1.1  Issues 

Common  issues  include: 

•  It  is  difficult  to  keep  track  of  vWorld  developments  as  there  are  many  which  in  turn  makes  it 
difficult  to  assess  developed  characteristics  that  may  be  useful  in  a  military  context 
(Maxwell,  2009).  The  underlying  software  architectures  of  vWorld  platforms  are  rarely 
disclosed  or  made  open  even  when  realized  using  Open  Source  Software  (OSS) 
components.  Thus,  it  is  a  major  technical  challenge  to  evaluate,  assess  or  compare  vWorld 
design  trade-offs.  This  is  true  whether  the  vWorld  has  been  developed  for  training  purposes 
(e.g.,  Project  Bluejacket,  see  Table  3-1)  or  used  to  support  a  collaborative  engineering 
activity  (e.g.,  potential  for  application  during  design  of  the  operations  room  layout  for  the 
Canadian  Surface  Combatant  ship).  Therefore,  a  framework  definition  is  required  to  help 
develop  open  architecture  (OA)  vWorlds  that  can  exchange  models  and  be  maintained  or 
expanded  efficiently  (Scacchi  &  Alspaugh,  2008;  Scacchi,  2012); 

•  There  is  a  lack  of  tools  to  support  real-time,  dynamic  prototyping  (creation  or  modification  of 
simulation  elements  while  the  simulation  is  running).  Maxwell  et  al.  (2011)  found  that  no 
vWorld  platform  had  capabilities  to  both  import  models  and  edit  those  models  in  real-time. 
Each  vWorld  that  was  examined  either  supported  mesh  model  importing  (hence  content  re¬ 
use)  or  supported  in-world  building  and  live  design  collaboration  in  a  proprietary,  geometric- 
primitive  format.  For  instance,  Maxwell  et  al.  (201 1 )  found  that  vWorlds  could  import  models 
into  the  environment,  but  options  for  modifying  and  adjusting  the  models  were  limited  or 
nonexistent  with  the  currently  available  vWorld  user  interfaces; 

•  Developing  a  secure  vWorld  that  can  be  used  on  public  networks  is  a  major  challenge  that 
will  require  collaboration  between  the  military  and  vWorld  industry  partners  (Aguiar  &  Monte, 
2011).  The  US  Military  Open  Simulation  Enterprise  Strategy  (MOSES7)  is  evaluating 
whether  OpenSIM  can  effectively  provide  secured  access  to  a  vWorld.  There  are  plans  to 
develop  secured/encrypted  communications,  user  authentication  with  certificates,  and 
integration/conformance  with  the  DoD  Virtual  World  Framework  (VWF).8  The  VWF  is  under 
development  to  work  with  vWorlds  such  as  OpenSIM  MOSES,  and  Massively  Multiplayer 
Online  Role  Playing  Games  (MMORPG)  such  as  EDGE.  Such  developmental,  secure 
environments  are  restricted  to  secure  networks  until  it  could  be  shown  that  they  could 
reliably  be  used  on  an  open  network;  and, 


7  MOSES.  Retrieved  on  05  February  2014.  http://brokentablet.arl.army.mil/ 

8  DoD  Virtual  World  Framework.  Retrieved  on  07  February,  2014: 
http://en.wikipedia.org/wiki/Virtual  world  framework 
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•  The  psychological  and  sociological  implications  of  moving  activities  (e.g.,  conferencing, 
training,  collaboration  and  experimentation)  into  an  immersive  environment  where  the  user 
is  perceived  first  by  their  avatar  and  not  their  physical  persona  are  largely  unknown  (Aguiar 
&  Monte,  2011).  More  research  is  required  to  determine  whether  there  are  adverse 
consequences  as  well  as  benefits  to  working  vicariously  through  an  avatar. 

4.1.2  Requirements 

The  focus  of  most  C2  related  work  has  been  on  the  user  interaction  and  the  collaboration 
aspects  of  vWorlds  (Aguiar  &  Monte,  201 1 ).  User  interaction  concerns  how  the  user  acts  in  the 
vWorld  and  how  the  vWorld  affects  the  user’s  avatar.  Collaboration  involves  what  is  needed  to 
help  stakeholders  (people  and  organizations)  achieve  shared  goals.  The  OpenSIM  platform 
provides  a  set  of  capabilities  that  can  facilitate  collaboration  within  an  immersive  environment, 
however,  much  work  is  required  to  create  functionality  that  would  be  desirable  by  military 
organizations  (Smith,  2011).  Capabilities  should  be  developed  by  employing  a  user  centred 
design  approach  that  could  conceivably  exploit  open-source  software  development. 

The  following  are  suggested  capabilities  derived  from  the  reviewed  literature  that  should  be 
provided  by  a  vWorld  to  support  collaboration  effectively.  It  is  unclear  whether  all  of  these 
recommendations  have  been  validated  or  are  merely  the  opinions  of  the  various  authors.  The 
basis  for  prioritizing  the  implementation  of  these  requirements  should  be  informed  by  the 
conduct  of  a  user  needs  analysis,  capabilities  assessment  and  cost-benefit  analysis  to 
determine  their  validity  unless  supported  by  the  scientific  literature: 

•  Allow  the  users  to  communicate  naturally  as  appropriate  to  the  application.  The  prevalent 
form  of  communication  between  group  members  is  verbal  conversations,  so  a  means  for 
verbal  communication  should  be  common  to  most  vWorlds,  although  apparently  many  only 
support  text  interfaces.  (Aguiar  &  Monte,  2011); 

o  Verbal  communications  options  may  include  technologies  such  as  Voice  over  Internet 
Protocol  (VoIP),  chat,  a  basic  level  of  multi-media  (e.g.,  streaming  video  onto  display 
surfaces),  virtual  aids.  The  objective  is  to  ensure  that  the  users  are  not  only  viewing  the 
information  but  are  part  of  the  information  space  (Maxwell  et  al.,  2011); 

o  Provide  the  means  for  gestural  communication.  Allow  explicit  gestures  and  other  visual 
actions  to  be  visible  since  they  are  done  in  direct  support  of  the  conversation  and  help 
convey  task  information.  There  are  cultural  differences  with  this  capability  that  should  be 
considered  to  reduce  misunderstanding,  particularly  with  international  participants; 

•  Include  the  means  for  remote  users  to  link  into  the  vWorld  via  cell  phones  (Tate,  Dalton,  & 
Potter,  2009); 

•  Provide  tools  in  the  user  interface  that  participants  can  use  easily  and  collaboratively  to 
modify  designs  or  the  virtual  environment  dynamically  in  real-time  (Aguiar  &  Monte,  201 1 ); 

•  Provide  a  means  of  protecting  the  work  of  multiple  users  editing  shared  models  in  real  time, 
providing  undo,  branching  and  model  recovery  (both  automatically  and  manually); 


24  June  2014 


24 


5457-002  Version  02 


CAE 

vWorld  Capability  Development  Support 
Literature  Survey 


•  Allow  stakeholders  to  review  the  evolution  of  design  decisions  (Aguiar  &  Monte,  201 1 ). 
Persistence,  with  links  to  source  material,  provides  knowledge  management  and  the 
rationale  behind  design  decisions.  The  use  of  OpenSIM  in  this  manner  would  enable 
DND/CAF  to  document  and  review  system  designs  such  as  compartment  layouts  and  team 
processes  on  naval  platforms; 

•  Models  should  be  maintained  as  technology  evolves.  Operating  systems  change  and  may 
no  longer  support  old  software  or  hardware  required  by  the  vWorld;  software  changes  may 
no  longer  support  old  model  definitions;  hardware  changes,  making  old  software  unusable; 
dependencies  may  be  broken  and  old  models  may  no  longer  function;  and, 

•  Provide  camera  controls  as  they  are  important  to  the  collaboration  process  (Maxwell  et  al., 
201 1 ).  Camera  controls  should  provide  the  means  for  users  to  quickly  shift  from  different 
perspectives  or  viewpoints  of  the  models.  Seeing  what  you  can  see  and  seeing  what  you 
think  others  can  see  is  important  in  collaborative  design  and  review. 

4.1.3  Practical  Considerations 

4.1. 3.1  Validation 

Modelling  and  Simulation  (M&S)  is  commonly  used  in  system  design  and  with  complex  systems. 
As  M&S  technologies  develop,  they  are  often  promoted  for  a  wide  range  of  purposes  with  little 
supporting  evidence  that  they  are  effective.  For  example,  Scipione  (2006)  conducted  an 
experiment  to  assess  different  display  technologies  for  detecting  design  flaws  in  computer 
assisted  design  environment.  User  performance  in  the  more  “advanced”  technologies,  such  as 
the  Data  Wall™  and  Cave™  displays  was  poor,  possibly  due  to  distracting  elements  of  the 
novel,  immersive  environments,  while  performance  on  more  traditional  2D  and  3D  displays  was 
superior  (although  each  had  differing  advantages).  Thus  validation  (fitness  for  purpose)  should 
not  be  assumed  as  “intuitively  obvious”  and  the  application  of  these  technologies  by  users  in  the 
entertainment  industry  should  not  be  considered  evidence  of  effectiveness;  similarly,  subjective 
opinions  of  usefulness  can  often  diverge  substantially  from  observed  performance  metrics  and 
realism  is  typically  an  ineffective  surrogate  for  validity.  M&S  often  deliberately  abstracts 
elements  to  facilitate  understanding  (Smallman,  2005;  Lintern  et  al.,  1997). 

Unfortunately,  there  is  insufficient  data  in  the  scientific  literature  to  make  generalizable 
conclusions  and  research  is  required  to  establish  the  range  of  validity  of  technologies  such  as 
OpenSIM.  Since  complex  system  designs  are  iterative,  such  as  in  Spiral  Development  (Boehm 
2000),  this  provides  opportunities  to  study  technologies  such  as  OpenSim  for  effectiveness  at 
various  stages  to  gain  an  understanding  of  where  it  is  effective,  where  it  is  not  effective  and 
where  it  is  cost-effective  relative  to  other  approaches.  Validation  research  can  follow  typical 
approaches  for  assessing  effectiveness  of  procedures  using  sound  experimental  design:  test 
the  vWorld  to  determine  if  it  can  produce  the  desired  result,  compare  the  result  to  alternative 
methods  (control  group  of  tradition  method;  alternative  technological  approach;  etc.),  and 
assess  the  relative  costs  of  the  successful  approaches. 

4.2  Training 

This  section  discusses  the  common  issues  and  requirements  of  using  the  OpenSIM  platform  for 
training.  Practical  considerations  are  then  proposed  to  address  these  issues  and  requirements. 
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4.2.1  Issues 

Training  issues  include: 

•  Compared  to  conventional  live  training  exercises,  simulated  scenarios  lack  the  direct  hands- 
on  experience  and  face-to-face  interactions  that  live  exercises  provide  (Hsu  et  al.,  2013). 
However,  live  training  is  expensive,  time-consuming,  and  is  not  easily  repeatable,  making  it 
difficult  to  assess  consistency  in  training  delivery  standards.  As  well  as  being  expensive, 
including  a  large  number  of  participants  for  live  training  may  not  be  practical.  There  is  an 
extensive  literature  on  the  successful  use  of  simulation  for  training  and  much  is  known  about 
learning  in  a  simulated  environment  (Salas  et  al.,  2009).  Unsuccessful  attempts  to  exploit 
computer  based  training  receive  less  exposure  in  the  literature,  although  there  are  a  few 
instances  that  demonstrate  the  need  for  scientific  validation  (Whitney  et  al.,  2013a;  2013b); 

•  Matching  training  requirements  to  vWorld  capabilities  is  a  challenging  endeavour.  A  level  of 
effort  is  required  first  to  understand  the  training  objectives,  i.e.  the  characteristics  of  what  the 
student  must  accomplish  within  the  vWorld  to  be  deemed  proficient  for  a  particular  role.  The 
functions  and  tasks  the  student  needs  to  perform  should  then  be  mapped  to  the  features 
and  capabilities  of  vWorld  platforms.  OpenSIM  may  not  be  the  best  platform  given  the  tasks 
that  need  to  be  performed  and  its  use  should  be  considered  within  an  instructional  design 
approach; 

•  The  topic  of  securing  a  game-based  vWorld  for  military  C2  applications  is  a  major  concern 
(Scacchi,  2012).  As  with  any  other  shared  networks,  there  are  risks  to  being  online  in  a 
shared  vWorld  and  most  vWorld  technologies  provide  only  very  basic  security  mechanisms. 
Furthermore,  game-based  vWorlds  may  allow  for  new  modes  of  malware  that  enable 
activities  including  avatar  impersonation  or  remote  control  (e.g.,  who/what  is  controlling  this 
avatar,  and  with  what  authorization?),  and  other  ill-defined  vulnerabilities  (Scacchi,  2012). 
Technological  solutions  such  as  Virtual  Private  Networks  (VPN),  local  private  networks  and 
distributed  leased  lines  may  be  used  to  address  security  issues,  as  is  current  practice  in 
other  domains; 

•  Designing  a  system  that  deliberately  shows  different  (and  incongruent)  views  of  the  vWorld 
to  different  users  is  problematic,  as  much  of  the  vWorld  work  has  focused  on  making  sure 
that  different  users  are  actually  present  in  the  “same”  virtual  space,  even  if  they  view  it  from 
different  viewpoints  (Carvalho  &  Ford,  2012).  The  design  of  a  vWorld  training  system  will 
likely  include  student  and  instructor  views  of  the  vWorld,  each  with  their  own  unique  set  of 
tools  to  interact  with  the  vWorld.  For  instance,  the  instructor  may  wish  to  control  playback  of 
the  vWorld  simulation  scenario,  assume  the  perspective  of  the  adversary  or  perform  after 
action  reviews  with  users.  OpenSIM  does  not  currently  offer  many  of  these  functions  and 
thus  may  not  be  the  right  platform  if  such  capabilities  are  required;  and, 

•  Collective  training  of  users  with  different  security  levels  (such  as  may  occur  with 
multinational  exercises)  poses  a  unique  challenge  for  military  applications  that  will  likely 
require  real-time  updates  on  information  release  and  access  control  policies  that  must  be 
enforced.  This  may  require  development  of  automatic  security  protocols  within  vWorlds  such 
as  OpenSim  that  may  not  have  adequate  level  of  user  management  or  even  the  software 
administration  tools  to  provide  user  management. 
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4.2.2  Requirements 

Training  in  vWords  must  support  the  following  elements: 

•  According  to  Maxwell  (2009)  the  vWorld  should  be  an  optimized  blend  of  simulation  and 
game  that  leads  the  learner  being  motivated  by,  and  immersed  into,  the  purpose  and  goals 
of  a  learning  interaction; 

•  Develop  and  refine  performance  metrics  that  can  be  applied  to  gauge  the  knowledge,  skills 
and  training  needed; 

•  The  vWorld  must  capture  the  essence  of  the  “operational  environment”  and  its  standard 
operating  procedures;  this  is  consistent  with  training  effectiveness  studies  that  show  high 
fidelity  representations  often  add  little  benefit  and  may  actually  impair  learning  because  of 
distracting  features  (Salas  &  Cannon-Bowers,  2001;  Salas,  Milham,  &  Bowers,  2003); 

•  An  adequate  user  experience  needs  to  be  supported  by  means  of  the  content  and  interfaces 
provided  (Tate,  Dalton,  &  Potter,  2009); 

•  Engagement  models  are  not  readily  available  in  open  source  vWorlds  so  development  effort 
would  be  required  to  build  such  models,  however,  experience  from  other  combat  modelling 
environments  may  be  applicable,  simplifying  development; 

•  Support  various  levels  of  fidelity  with  respect  to  the  number  and  nature  of  participants  and 
the  technology  mediating  the  training; 

•  Ensure  course  material  is  targeted  appropriately  to  individuals  within  the  audience  as  well  as 
to  overall  teams  that  participate  within  training  environments;  and, 

•  Provide  a  means  to  apply  or  relax  the  laws  of  physics  of  the  vWorld  application  (Allen  & 
Demcahk,  2011). 

4.2.3  Practical  Considerations 

4.2.3. 1  Directed  Investigation  of  Training  Effectiveness 

Training  is  a  complex  endeavour  that  sometimes  displays  counter-intuitive  results.  Effective 
training  typically  requires  a  disciplined  application  of  instructional  design,  applying  what  is 
known  about  how  people  learn,  how  learning  can  be  disrupted  and  how  technologies  support 
learning.  There  is  a  generally  held  impression  that  immersive  environments  that  are  realistic  and 
engaging  are  better  learning  environments  but  that  is  a  simplistic  perspective.  There  is  a 
substantial  literature  on  the  usefulness  of  simulation  technology  that  supports  learning,  but  the 
use  of  vWorlds  for  training  is  still  in  its  infancy.  There  is  some  empirical  evidence  that  vWorlds 
may  be  useful  (Bailenson  et  al  2006,  2008,  Dede  2009)  but  generally  the  effect  sizes  are 
modest  to  small.  Nevertheless,  vWorld  technologies  such  as  OpenSIM  can  be  reasonably 
expected  to  be  both  effective  and  affordable  in  many  situations  provided  they  are  applied  in  an 
informed  manner  following  instructional  design  principles  and  the  wealth  of  scientific  literature 
on  instruction  simulation  technologies  is  considered. 
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4.2. 3.2  Validation 

Validation  of  a  vWorld  for  learning  and  training  can  exploit  a  few  approaches  (AGARD  1980), 
but  two  in  particular  have  been  applied  to  validation  of  simulators.  Both  involve  assessment  of 
whether  it  is  reasonable  to  expect  that  what  is  learned  in  the  vWorld  will  transfer  to  the  real 
world,  although  the  approaches  differ.  The  first  is  the  gold  standard:  Forward  Transfer  of 
Training.  This  involves  assessing  a  training  program  with  a  cohort  of  subjects  in  the  simulator  (in 
this  case,  the  vWorld)  and  then  putting  the  training  cohort  into  the  real  world  situation.  A 
measure  of  performance  is  the  learning  curve  and  learning  rate  of  the  group,  and  whether  the 
vWorld  contributes  to  the  learning  curve  advantageously.  Comparison  with  learning  in  the  real 
world  situation  is  frequently  used  as  a  control  and  both  effectiveness  and  cost-effectiveness 
should  be  considered  when  judging  the  overall  performance  of  the  intervention.  A  second 
approach,  Reverse  Transfer  of  Training,  does  not  provide  as  definitive  an  answer,  but  may  be 
an  effective  alternative  when  transferring  the  training  cohort  to  the  real  world  situation  is 
expensive  or  risky.  In  the  Reverse  Transfer  of  Training  approach,  the  learning  curve  of  a 
learning  cohort  is  compared  to  that  of  a  domain  expert  cohort  and  the  differences  are  used  to 
predict  training  effectiveness  of  the  (vWorld)  training  intervention. 

Aguiar  and  Monte  (201 1 )  seemed  to  apply  the  Forward  T ransfer  of  T raining  method  to  validate 
the  performance  of  fleet  operators  in  a  vWorld  compared  to  a  physical  world.  The  experiment 
involved  two  teams  of  fleet  personnel  performing  submarine  Target  Motion  Analysis  (TMA)  to 
identify,  classify,  and  track  a  contact  of  interest.  Each  team  ran  through  a  TMA  scenario  twice  - 
one  trial  allowed  the  operators  to  access  the  submarine  combat  system  from  the  actual 
Command  and  Control  Center  (CACC)  hardware,  and  the  other  trial  allowed  access  to  the 
tactical  systems  via  the  vWorld  CACC.  While  the  number  of  participants  in  the  study  cannot  be 
considered  statistically  significant,  the  study  has  revealed  a  number  of  lessons  learned.  The 
results  of  this  experiment  suggest  that  operators  performed  TMA  equally  well  in  both  trials.  The 
results  also  suggest  that  C2  operator  activities  from  within  a  remote,  distributed  virtual 
environment  are  comparable  (and  thus  measureable)  to  operation  in  the  real  environment. 
Similar  experiments  should  be  conducted  with  a  larger  number  of  participants  in  future 
experiments  to  assess  team  performance  and  vWorld  performance  compared  to  a  baseline. 
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5  CONCLUSIONS  &  RECOMMENDATIONS 

The  conclusions  of  this  work  are: 

•  Several  vWorld  platforms  are  available  that  could  be  and  are  being  applied  to  military 
domains  (at  least  on  an  experimental  basis)  including:  Second  Life,  OpenSIM,  Virtual  Battle 
Space  2  (VBS2)  /  VBS3,  Active  Worlds,  Croquet,  On-Line  Interactive  Virtual  Environment 
(OLIVE),  Open  Colbalt  Alpha,  OpenEnergySim,  Open  Wonderland,  Pivote  and  Teleplace; 

•  Converging  trends  reveal  that  OpenSIM  is  being  demonstrated  as  a  capable  and  flexible 
platform.  There  is  an  active  OpenSIM  user  community  in  the  application  domains  of  interest 
to  DRDC  (Section  3.2)  including  organizations  such  as:  US  NUWC,  DSTO,  the  US  Navy 
Postgraduate  School,  Florida  Institute  of  Technology  (Harris  Institute  for  Assured 
Information),  University  of  Edinburgh  (Artificial  Intelligence  Applications  Institute),  and  the 
US  Army; 

•  OpenSIM  can  support  diverse  applications  including  systems  development  and  training,  and 
the  number  of  institutions  exploring  vWorlds  in  general  seems  to  be  increasing  as 
awareness  of  the  technology  increases.  However,  the  scientific  literature  assessing  the 
validity  of  vWorlds  for  training  or  analysis  is  immature  and  the  effectiveness  of  these 
technologies  remains  to  be  determined; 

•  Research  and  development  or  at  the  very  least  knowledgeable  analysis  is  needed  to  match 
training  requirements  to  evolving  vWorld  capabilities  in  order  to  establish  the  validity  of  a 
vWorld  for  the  intended  training  application.  Much  of  what  is  known  about  the  effectiveness 
of  simulation-based  training  should  be  applicable,  but  there  are  many  misconceptions  held 
by  education  practitioners; 

•  Basic  research  is  needed  to  develop  open  architecture  (OA)  frameworks  for  specifying 
vWorlds  to  promote  exchange  and  reuse  of  models  as  well  as  reduce  the  incidence  of 
models  that  cannot  be  exchanged  between  virtual  environments  (Scacchi  &  Alspaugh,  2008; 
Scacchi,  2012);  and, 

•  OpenSIM  can  be  linked  to  other  analysis  tools  through  an  Application  Programming 
Interface  (API). 

Recommendations  are: 

•  Employ  a  user  centred  design  approach  to  develop  tools  fit  for  use  in  vWorlds  for  the 
purpose  of  systems  development  and  training.  In  accordance  with  guidance  from  MIL-STD- 
46855A,  the  Mission,  Function,  and  Task  Analysis  (MFTA)  methodology  has  been  widely 
used  within  the  Human  Factors  community  to  conduct  analyses  of  user  requirements  for 
over  twenty  years.  Suggested  capabilities  derived  from  the  reviewed  literature  that  should 
be  provided  by  a  vWorld  to  support  systems  development  and  training  effectively  are 
documented  in  Section  4.1.2  and  Section  4.2.2,  respectively; 

•  Conduct  a  number  of  use-case  experiments  that  compare  vWorld  development  to 
conventional  design  methods  to  assess  the  relative  costs  and  effectiveness  of  immersive 
environments  for  design.  The  use-cases  should  be  sufficiently  challenging  that  they  can 
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discriminate  between  the  design  approaches,  but  limited  in  scope  so  that  they  could  be 
conducted  in  an  affordable  manner.  Further,  the  experimental  design  should  explicitly 
include  adequate  training  in  both  traditional  and  vWorld  design  methods  to  minimize  biases 
due  to  learning  the  methods  per  se  (Scipione,  2006); 

•  Employ  a  disciplined,  instructional  design  approach  before  considering  vWorlds  as  a 
potential  solution  for  training  delivery; 

•  Conduct  a  directed  investigation  of  training  effectiveness.  Effective  training  typically  requires 
a  disciplined  application  of  instructional  design,  applying  what  is  known  about  how  people 
learn,  how  learning  can  be  disrupted  and  how  technologies  support  learning;  and, 

•  Validate  the  use  of  OpenSIM  for  learning  and  training  by  employing  transfer  of  training 
analysis  methods  as  prescribed  by  AGARD  (1980). 
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7  LIST  OF  ACRONYMS 


2D 

2-Dimensional 

3D 

3-Dimensional 

4D 

4-Dimensional 

AIS 

API 

AWEDU 

Automatic  Identification  System 

Application  Programming  Interface 

Active  Worlds  Educational  Universe 

C2 

CACC 

CAF 

CATIA 

CFB 

COLLADA 

COOPEX 

CVE 

Command  and  Control 

Command  and  Control  Center 

Canadian  Armed  Forces 

Computer  Aided  Three-dimensional  Interactive  Application 

Canadian  Forces  Base 

Collaboration  Design  Activity 

Concept  of  Operations  Exercise 

Collaborative  Virtual  Environment 

DCC 

DECENT 

DLSE 

DND 

DoD 

DRDC 

DSTO-AUS 

DWAN 

Decentralized  Command  and  Control 

Decentralized 

Director  Land  Synthetic  Environments 

Department  of  National  Defence 

Department  of  Defence 

Defence  Research  &  Development  Canada 

Defence  Science  and  Technology  Organisation  -  Australia 

Defence  Wide  Area  Network 

ERP 

Enterprise  Resource  Planning 

FPS 

First  Person  Shooter 

HE 

HSI 

Human  Engineering 

Human  Systems  Integration 

iBAL 

IDE 

IPT 

1-Room 

ITS 

Unknown 

Integrated  Development  Environment 

Integrated  Product  Team 

Intelligent  Room 

Intelligent  Transportation  System 

JIMP 

Joint  Interagency  Multi-National  and  Public 

M&S 

MFTA 

MMORP 

MOSES 

MPAT 

Modelling  and  Simulation 

Mission  Function  Task  Analysis 

Massively  Multiplayer  Online  Role  Playing  Games 

Military  OpenSIM  Enterprise  Strategy 

Multi-National  Planning  Augmentation  Team 
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MPML3D 

Multi-modal  Presentation  Markup  Language 

NASA 

NS 

NUWC 

National  Aeronautics  and  Space  Administration 

Nova  Scotia 

Naval  Undersea  Warfare  Center 

OLIVE 

OpenSIM 

Ops 

On-Line  Interactive  Virtual  Environment 

Open  Simulator 

Operations 

PDF 

Portable  Document  Format 

SA 

SL 

SOW 

Scientific  Authority 

Second  Life 

Statement  of  Work 

TMA 

Target  Motion  Analysis 

US 

USB 

USW 

United  States 

Universal  Serial  Bus 

Under  Sea  Warfare 

VBS2 

VBS3 

VE 

VoIP 

VPN 

VW 

vWorlds 

VWF 

VWT 

Virtual  Battlespace  2 

Virtual  Battlespace  3 

Virtual  Environment 

Voice  over  Internet  Protocol 

Virtual  Private  Network 

Virtual  World 

Virtual  Worlds 

Virtual  World  Framework 

Virtual  World  Technology 

XML 

Extensible  Markup  Language 
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APPENDIX  A  ADDITIONAL  INFORMATION 
A.1  Search  Results  (Original  Scope) 

This  section  presents  the  search  results  for  the  original  scope  of  work  divided  into  categories 
based  on  whether  they  were  subsequently  approved  for  review  or  not.  Most  are  presented  as 
links  to  the  documents. 

A.1.1  Approved  for  Review:  High  Priority  Items 

US  Army's  Military  Open  Simulator  Enterprises  Strategy  (MOSES)  grid 

•  EdMedia  -  Military  Open  Simulator  Enterprise  Strategy 

•  Military  Open  Simulator  Enterprise  Strategy  Components  and  Architecture  12.JUN.12 

•  MOSES  MAP:  Interactive  region  grid  map  with  panning  and  zoom 

OpenSim  Grid  List:  A  community  generated  list  of  virtual  worlds. 

Aguiar,  S.,  &  Monte,  P.  (2011).  Virtual  worlds  for  c2  design,  analysis,  and  experimentation. 

Naval  Undersea  Warfare  Center  Division,  Newport,  Rl. 

Boldyreff,  C.,  Dastbaz,  M.,  Liu,  H.,  &  Arafa,  Y.  (2011).  Engineering  advanced  training 

environment  for  crisis  management:  The  Pandora  project.  In  Proceedings  of  Advances  in 

Computing  and  Technology  Conference ,  pp.  125-131. 

Bowman,  C.  M.,  Lake,  D.,  &  Hurliman,  J.  (2010).  Designing  Extensible  and  Scalable  Virtual 

World  Platforms.  In  Extensible  Virtual  Worlds  Workshop  (XI 0). 

Carvalho,  M.  M.,  &  Ford,  R.  (2012).  NextVC2  -  A  next  generation  virtual  world  command  and 

control.  In  Military  Communications  Conference,  2012,  pp.  1-6.  IEEE. 

Champsas,  I.,  Leftheris,  I.,  Tsiatsos,  T.,  Terzidou,  T.,  &  Mavridis,  A.  OpenGames:  A  Framework 

for  Implementing  3D  Collaborative  Educational  Games  in  OpenSim.  In  6th  European 

Conference  on  Games  Based  Learning ,  p.  82.  Academic  Conferences  Limited. 

Cohen,  D.,  Sevdalis,  N.,  Patel,  V.,  Taylor,  M.,  Lee,  H.,  Vokes,  M.,  ...  &  Darzi,  A.  (2013).  Tactical 
and  operational  response  to  major  incidents:  feasibility  and  reliability  of  skills  assessment  using 
novel  virtual  environments.  Resuscitation. 

Dafli,  E.  L.,  Veqoudakis,  K.  I.,  Pappas,  C.,  &  Bamidis,  P.  D.  (2009).  Re-use  and  exchange  of  an 

Opensim  platform  based  learning  environment  among  different  medical  specialties  for  clinical 

scenarios.  In  9th  International  Conference  on  Information  Technology  and  Applications  in 

Biomedicine ,  2009,  pp.  1-5.  IEEE. 

Deeds,  D.  (2011,  March).  Establishing  3D  Virtual  World  Presences  for  the  International 

Baccalaurate  Organization's  Teachers,  Administrators  and  Students.  In  Global  Learn  vol.  2011, 

no.  1,  pp.  901-901. 
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Delp.,  S.  L.,  Anderson,  F.  C.,  Arnold,  A.  S.,  Loan,  P.,  Habib,  A.,  John,  C.  T.,  Guendelman,  E. 

and  Thelen,  D.  G.  (2007).  OpenSim:  Open-Source  Software  to  Create  and  Analyze  Dynamic 

Simulations  of  Movement.  IEEE  Transactions  on  Biomedical  Engineering ,  54  (11),  pp.  1 940- 

1950. 
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Second  Life  and  OpenSim.  Educause  Australasia. 
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